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Abstract 

This study tracks initial effects of a philanthropic effort, while disentangling its effects from a 
major concurrently administrated school improvement initiative. Significant positive effects 
attributable to both efforts were noted in a middle school student sample. The full impact of the 
philanthropic and school improvement initiatives will manifest over the years as more and more 
students mature and matriculate through the system. As we continue to track and monitor student 
cohorts, we plan to examine the impact on dropout rates, school performance, and attendance and 
to attempt further to disaggregate school improvement efforts from those of the philanthropic 
effort. 
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Disentangling School Reform Outcomes from School Philanthropy Influences: 
The Kalamazoo Promise Two Years Later 



Ever since A Nation at Risk, the report produced in 1983 by the National Commission of 
Excellence in Education, schools have been implementing a range of school reforms from the 
Coalition of Essential Schools whose mission is to create learner-centered schools (Ancess, Ort, 
1999) to professional learning communities (Fullan, 2006) that focus on job-embedded professional 
development to small learning communities that personalize the school experience for students 
(Rourke, Mero, 2008). With the advent of No Child Left Behind (2002), the concept of data-driven 
decision making, which espouses continuous monitoring and improvement of student achievement 
through the use of data collection, analysis, and feedback , has influenced school reform (Bernhardt, 
2004). Concurrent with school reform and despite the economic downturn, this nation has 
experienced added wealth over the past several decades. This has presented opportunities for dollars 
to be funneled into philanthropic support for school reform (Born & Wilson, 2000). Philanthropists 
interested in demonstrating proactive school reform results (Anrig, 2008) have targeted their 
resources toward creative efforts aimed at promoting not only school reform (Domanico, Innerst, & 
Russo, 2000) but also socio-economic integration (Mirron, Evergreen, 2007). In this paper, we 
attempt to answer the following questions: given the dollars that have been funneled into both 
school reform and philanthropic efforts, how does one disentangle the effects of one from the other 
when both are occurring simultaneously within the same building? In what ways do school reform 
and philanthropy help and/ or hinder the goal of increasing student achievement? 



To accomplish this, we have disentangled the effects of a larger economic philanthropic 
community based scholarship program, the Kalamazoo Promise (KP) that was implemented in 2005 
from concurrent effects of a multi-million dollar, multi-year, federally-funded Midwest Educational 
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Reform Consortium / Gaining Early Awareness and Readiness for Undergraduate Programs (GU) 
grant which also began in the fall of 2005. This study examines the trajectory of marking period 
grade point averages of students in three sixth grade cohorts at two middle schools from 2006 
through 2008, before and after the announcement of the KP scholarship. Comparative data is 
presented for similar student groups in a neighboring school district also served by GU concurrently 
and for the previous 5 years but not part of the KP initiative. 

The School District 

Kalamazoo Public Schools (KPS) is a mid-size urban school district in Michigan with a 
student population of about 11,500, of which 63 percent are students of color. The district has 17 
elementary schools, one of which is a dual language school, 3 middle schools and will be opening a 
fourth middle school in the fall of 2009, an alternative middle school program, two high schools, 
and an alternative high school. In the two middle schools involved in this study, 71 percent of the 
students received free or reduced lunch, a 22 percent increase since 2004. This rise in poverty has 
been fueled by the significant loss of jobs through corporate mergers and down-sizing in the 
banking, pharmaceutical, and paper manufacturing industries. KPS was one of only nine districts in 
the state (493) that fell below state expectations for academic progress in elementary, middle, and 
high school and both high schools were required to submit a restructuring plan because of under- 
performance in standardized tests for NCLB in 2006. Only 56 percent of incoming freshmen 
graduate in 4 years, over 56 percent of the incoming 2007 freshmen class are 3 grade levels behind 
in reading and math, and 39 percent of the twelfth grade students fail to meet state standards. 

The comparison district is a smaller urban district with 7125 students. From 2001 to 2006, 
this district has seen a decline in enrollment of 1000 students, a 12 percent decrease, causing 
considerable financial impact and the closing of four schools. However, during this period of time, 
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these middle schools have benefitted from a major GU grant focused on school-based reform. 
Currently, more than 75 percent of the students in this district qualify for free/reduced lunch, a 20 
percent increase from 2004. Approximately 54 percent are children of color. 

Student achievement is of particular concern as the percentage of students meeting state 
standards in math has been steadily declining from 1999-2006 in spite of improvement efforts but 
concomitant with national patterns. In the high school there has been a decrease of 12 percent in 
math. In addition, there was an 1 1 percent decrease in middle school writing scores over the same 
period of time. Significant gaps in achievement have also been identified between White, Hispanic, 
and Black students; between economically advantaged and economically challenged students; and 
between general education and special needs students. On average each year, only 56 percent of the 
incoming freshman will graduate. Of those who graduated in 2007, less than 37 percent have plans 
to go on to college. 

Both KPS and the comparison school district face a number of relevant and similar issues. 
Many demographic issues as well as student attendance and drop-out rates have adversely affected 
enrollment. In addition, academic achievement and behavior are an ever present challenge in these 
urban schools. 



School Reform Efforts 

The current GU reform grant is a multi-year, multi-million dollar grant funded in part by the 
U.S. Department of Education that targets cohorts of sixth through twelfth grade students in 
schools that serve children of poverty. The purpose of the grant is to help provide stmctures and 
support for a rigorous, standards-based curriculum that prepares students for post-secondary 
education. The GU project, through a school-university partnership, is broadly adaptive to the 
unique needs of individual schools and communities. As such, GU funded school reform projects 
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target specific need areas of the affiliated schools as identified by local school administrators. The 
project is based upon a systematic focus on school reform, professional development, 
student/ family support, and community engagement in order to prepare students socially and 
academically for post-secondary education fwww.wmich.edu/geamp). Currently, the GU project 
encompasses four school districts in two states. The specific focus of this study centers on students 
enrolled in two middle schools in the Kalamazoo Public School District, Michigan, (KPS) and in 
middle schools in a comparison district who entered GU cohorts in the 2005-2006 and 2006-2007 
academic years and who have since transitioned to a common high school. 

Philanthropic Efforts: The Kalamazoo Promise 

The KP was announced during the same year (2005-2006) that KPS became involved with the 
GU grant-funded school-university partnership. At the end of the 2005-2006 academic year and 
subsequent years, qualifying KPS graduates were eligible to have up to 100 percent of their college 
tuition to any Michigan university and/ or community college paid for by this scholarship fund. 
Students who did not attend all K-12 years at KPS were offered prorated scholarship benefits 
(www.kalamzoopromise.com). Thus, since 2005, qualifying KPS students were able to take 
advantage of the federally funded school improvement effort (GU) and anonymously funded 
scholarship program (KP). 

The scholarship is not based on economic or academic need, but rather on the geographic 
location of residency (www.upjohninst.org/promise). The purpose of the KP is economic 
development that “seeks to revitalise the city [of Kalamazoo] and the region through a substantial investment in 
public education (www.upjohninst.org/promise, para.2). According to the KPS website, the KP 
scholarship “is a catalyst to [make] the Greater Kalamagoo Community an even greater community and is inspiring 
KPS students, families, teachers and staff. It is re-energising the citisens of Kalamagoo and the Greater Kalamagoo 
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community ( wmv.kalamayoopromise.com ). The KP scholarship, therefore, is an investment focused on 
promoting socio-economic integration by paying for college tuition (www.upjohninst.org/promise). 
The only program element defined by the KP is the actual scholarship. For example, tuition and fees 
at Western Michigan University for freshman entering the class of 2013 are projected to be $33,400. 
Long and short term outcomes and possible impacts on the school environment and student 
achievement are delineated in the evaluation component of the KP, which is being conducted by the 
W.E. Upjohn Institute in Kalamazoo and the Evaluation Center at Western Michigan University. 

Disentangling School Reform from Philanthropy 

If students are to realize the opportunities for post-secondary education provided by the KP, 
then other factors must be in place before graduation, e.g. evidence of student achievement 
commensurate with college entrance expectations. This is the primary outcome of GU. Research is 
clear with regard to what factors are necessary for academic success, graduation, and post-secondary 
education involvement to occur (Ancess, Ort, 1999; Fullan, 2006; Rourke, Mero, 2008). These 
factors are contained in the overall goals and objectives of the GU project. The GU goal is to 
increase the number of low income students who are prepared to enter and succeed in post- 
secondary education. The objectives of the project are to increase yearly academic performance and 
preparation for post-secondary education, to increase the rate of high school graduation and 
participation in post-secondary education, and improve students’ and their families’ knowledge of 
post-secondary options, preparation, and financing (www.ed.gov/programs/gearup). 

Clearly both GU and the KP will impact students and their future plans for post-secondary 
education. GU focuses on affecting student achievement via school reform and cohort support 
while KP provides the necessary financial foundation for all students within the KPS District to 
attend a college or university. 
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But if the student is not academically prepared for the rigors of post-secondary education, the 
larger economic goals of the KP will not be realized. Research studies on school reform address a 
number of factors that are relevant for school academic success including interdisciplinary 
curriculum (Wiggins & McTighe, 2005), a caring/ collaborative learning environment (Brophy, 

1988), job-embedded professional development (Huffman, et al, 2001), and professional learning 
communities (Senge, 2006), all of which are featured as GU interventions in KPS and the 
comparison district collaborations. Other studies have focused on the effect of the KP on the 
perceptions of community leaders, parents, students, and school personnel (Miron, et al, 2007, 

2008). Few studies, however, attempt to relate the effects of a private philanthropic effort on 
student achievement or to disentangle these effects from those of other initiatives. As further 
longitudinal studies of the KP progress over ensuing years, one must keep in mind the caveat that 
“because education philanthropy is disorderly and little studied but highly visible, it is easy both to 
overestimate and to underestimate its significance" (Hess, 2005, p.132). 

Sample Participants and Research Design 

This study began with a cohort of sixth grade students in two KPS middle schools in the 2005- 
2006 school year (N=256) and followed these students for 3 school years and 6 semesters. KPS 
operates on a 6 marking period calendar while the comparison district operates on 4. Semester total 
GPA is the average of 3 marking periods in KPS and 2 marking periods in the comparison district. 

A second cohort of sixth graders (N=237) entered in the school year ending in 2007 and is followed 
for 4 semesters through 2008. A third cohort (N=l 10) entered in 2008 and has been followed for 2 
semesters at the time of this writing. By convention, we refer to an academic year by ending year 



number. 
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As part of the effort to disentangle intervention effects, it should be noted that while the 
KPS/ GU collaboration had just begun in 2006 and had been underway one year by 2007 and two 
years by 2008 and while school change and innovation continued to occur, none of the students in 
each incoming cohort had any prior experience with GU and thus were equivalent in this respect. 

On the other hand, while the comparison school district had benefitted from 5 prior years of 
GU/comparison district collaboration and school reform, entering sixth graders in the three cohorts 
there also had no prior experience with GU and were similar to each other and to the KZ cohorts in 
terms of demographics and prior academic preparation for middle school. Further, the new 
KPS/GU and on-going comparison district/GU collaborations now are primarily cohort-based 
interventions focusing on particular groups of students moving through the educational system 
(sixth through 12th grade), rather than school-reform collaborations with students flowing in and 
out interchangeably. In school years ending in 2006 and 2007, project cohorts comprised all sixth 
and seventh graders in the target schools served by GU. These students were followed as they 
became seventh and eighth graders in 2007 and 2008. The sixth graders in 2008 are not formally part 
of the GU cohorts and did not receive direct services from the project. Therefore, the feature 
distinguishing the two districts is 5 years of GU school reform in the comparison district and zero 
years in KPS. The feature distinguishing the three cohort groups in both districts is the amount of 
time elapsed since the public announcement of the KP. This occurred in the middle of the first 
cohort's sixth grade year and they and their families had essentially zero advance notice. By the time 
the second cohort entered sixth grade, the KP had been discussed publicly as a major community 
innovation and opportunity for over 9 months. By the time the third cohort entered sixth grade, KP 
had been in operation almost two years and was well understood and taken for granted. 

A similar group of three sixth grade cohorts entering four middle schools in 2006 and 2007 
and 2008 academic years (N=500, 467, and 378 respectively) in a neighboring city with similar 
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characteristics was followed as a comparison group. This school district operates on a 4 marking 
period per academic year system and these cohorts were served similarly by the GU project. 
However, this district does not have a comparable philanthropically funded, system-wide general 
scholarship incentive program and, in this respect, can serve as an appropriate comparison group. 
Table 1 illustrates the sampling model, showing students in the two districts and three sixth grade 
cohorts being observed for 6, 4, and 2 semesters respectively over a three year period. Table 2 
presents information on the categorical variables (factors) in the study and number of observations 
pooled at each level of each main effect. 



Insert Table 1 and 2 about here 



Analytical Methodology 

The academic performance of the KPS and comparison student groups was modeled using 
generalized estimating equations (GEE) with a first-order auto-regressive [AR(1)J variance- 
covariance stmcture in a linear-normal AN OVA model appropriate for a causal-comparative, 2- 
between, 1 -within univariate repeated measures design. The dependent variable defining academic 
performance was total semester grade point average (GPA), for both core and non-core coursework 
attempted. Student GPA is considered to be a function of individual within student variation, 
district differences due to prior GU experience (two fixed effect levels), and elapsed time following 
public announcement and inception of the KP (three fixed effect levels: 0, 1 year, and 2 years) 
measured across six elapsed semester grading periods (within subjects effect). 
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Results 



A summary overview of key results from the study is presented in Figures 1 and 2 with 
supporting statistical results shown in Table 3. Overall group mean differences between the cohorts, 
controlling district differences and semester to semester changes were not significant (Wald X 2 2 = 
.318, p = .853). This is something the school system and GU can be proud of because at a time and 
a grade-level/ age -level range where low income student cohorts are often declining in academic 
performance regionally and nationally, these student groups in both Kalamazoo and the comparison 
district maintained a fairly consistent and adequate level of academic performance throughout 
middle school. 



Insert Table 3 about here 



Insert Figure 1 and Figure 2 about here 

However, the main effect for group was overshadowed by the highly significant within- 
subjects effects to be noted as a main effect over time (Wald X 2 = 267.65, p < .001) and a two and 
three way interaction between time, school district, and cohort group (see Table 3). The positive 
effect of deliberate programmatic intervention (the GU effect for five prior years in the comparison 
district) predominates in Figure 1 which diagrams the district by elapsed time interaction, controlling 
for cohort effects (Wald X 2 = 21.211, p < .001). The differences between the cohorts (which differ 
primarily in terms of familiarity and confidence in the KP compared to status quo in the comparison 
sample where there was no such effect) can be seen in Figure 2 as patterns change over time, with a 
major bump in the performance of the KPS 2006 cohort following the announcement (Wald X 2 5 = 
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21.211, p < .001). This, we argue, is early evidence for an evolving, confidence-building attitude 
change toward the possibility of college reflected in better student academic performance in school. 
Unfortunately, this social philanthropic effect appears to fade as time goes by and the KP becomes 
well-understood and assumed as part of the social milieu. 

Additional support for this inference is provided by the comparative analysis of similar events 
and consequences in the neighboring school district without benefit of a program like the KP. The 
significant group difference trends during comparable time periods noted above as well as significant 
time effects and interactions could be due to grade inflation, elementary school preparation for 
middle school, and many other factors at work. But the trends over time and the changes in group 
differences over time in this school district are more downward sloped and more closely resemble 
other regional and national statistical patterns. In addition to the difference in prior experience with 
GU, another general significant public and political difference between these two school districts 
and these students over the timeframe of this study is the existence of the KP initiative. This effect 
is still apparent in Figure 2 for KPS students two semesters out from the time of the announcement. 

Conclusion 

This study attempted to disaggregate effects of a major programmatic educational 
intervention and the effects of a major social philanthropic intervention on early academic 
performance of middle school students in one school district over a period of three years' 
observation. This could be done because of the availability of comparable data from a second similar 
school district and city also served prior to and concurrent with the district where the philanthropic 
intervention occurred. 

KPS and the comparison district are similar in that they have many neighborhoods, families, 
and children presenting many challenges to traditional public schools. These district schools were 
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having great difficulties meeting these various challenges, which is why they (and the students) met 
the requirements to be GU schools. While these districts and their cohorts are similar, the 
comparison district benefitted gready from five previous years of GU school reform, including 
teacher professional development, whereas KPS was just starting out with GU in 2006. 

Our results show a substantial GU effect: two similar school districts, one after 5 years of 
school improvement efforts and the other at point 0. The comparison district students performed 
much better across the board and over the middle school years than did the KPS students. In 2006, 
both KPS and the comparison district began receiving cohort-based services rather than school 
reform services. As we watch the cohorts over time, there is an unfortunate but expected decline 
evident in both places, greater at first in KPS but eventually after several years the KPS cohorts 
match the comparison district cohorts who resemble similar nadonal populations in regard to this 
trend. 



The KP was announced at the same time GU started in KPS. There was a short blip of even 
greater improvement that appears to have faded somewhat i.e., the effects of philanthropy. We try to 
separate all these effects in this study and anticipate as the project continues that the good effects of 
the KP in KPS and of GU in KPS and the comparison district will replicate with current and new 
cohorts as these student progress into high school and beyond. 
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Table 1: Sample Model 



KZ Promise Announced 



School Year 


2006 
SI S2 


2007 
S3 S4 


2008 
S5 S6 


District 


Cohort 


KZ 


Cl 


kz 61 kz 62 


KZ 71 kz 72 


N 

CO 

N 

oo 


C2 




kz 61 kz 62 


kz 71 kz 72 


C3 






kz 61 kz 62 


BC 


Cl 


bc 61 bc 62 


bc 71 bc 72 


bc 81 bc 82 


C2 




bc 61 bc 62 


bc 71 bc 72 


C3 






bc 61 bc 62 
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Table 2: Categorical Variable Information 



Factor 


Level 


NObs 


Percent 


District 


BC 


4901 


67.1% 




KZ 


2407 


32.9% 




Total 


7308 


100.0% 


Cohort 


2006 


3866 


52.9% 


Starting 


2007 


2506 


34.3% 


in Year 


2008 


936 


12.8% 




Total 


7308 


100.0% 


Elapsed 


SI 


1888 


25.8% 


Semesters 


S2 


1872 


25.6% 


from Start 


S3 


1216 


16.6% 


of 6th Grade 


S4 


1182 


16.2% 




S5 


584 


8.0% 




S6 


566 


7.7% 




Total 


7308 


100.0% 



Table 3: ANOVA Summary Table 



Source 


5 

Wald Chi- 
Square 


[ype III 
Df 


Sig. 


(Intercept) 


10180.220 


1 


.000 


District 


3.769 


1 


.052 


CYR 


.318 


2 


.853 


elapsem 


267.650 


5 


.000 


District * CYR 


15.102 


2 


.001 


District * elapsem 


21.211 


5 


.001 


CYR * elapsem 


9.230 


4 


.056 


District * CYR * elapsem 


15.599 


4 


.004 



Dependent Variable: Term TOTAL GPA 

2B1W repeated measures linear GEE Model using AR1 
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Average Student Academic Performance Patterns 
OverTime From 6th Grade by District 
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Figure 1. Two-Way Interaction for District by Elapsed Time Within Students 
Illustrating the GEAR UP Experience Effect 
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Average Student Academic Performance Patterns 




Figure 2: Three-Way Interaction for District by Cohort by Elapsed Time Within 
Students Illustrating the KZ Promise Effect 



